Back

31/07/23 - Permissions API

Deploying.

27/07/23 - Permissions API

Finishing permissions API.

Reorganising structure of api services on gitlab.

26/07/23 - Permissions API

Working version.

25/07/23 - Permissions API

Made a start on porting the permissions work to a single api.

24/07/23 - Datalink

Added async endpoint to datalink response.

Fixed issue in site-directory with "other_attributes" field not populating.

Helped sort Calico network policy issue for dask hub.

Finished adding RAC dataset to testing_ingest_180723 scope (688 files w/ metadata).

TODO:

20/07/23 - Fixing datalink response

Added logic to construct template for SODA.

IVOA authority is as specified by orange otherwise service doesn't work.

19/07/23 - Support / meetings

Redefining schema for services to site-directory to enable SODA / ingest.

Added public /services endpoint for sites.

Negotiating SODA mounting, now works e.g. curl --get --data-urlencode "ID=ivo://auth.example.org/datasets/fits?sp3531_soda/9b/98/2023-07-18-16-40-27_ASK-WALLABY_Eridanus_cutout-574594-imagecube-42178.fits" --data-urlencode "CIRCLE=53.753 -24.933 0.1" -v -o soda.fits http://spsrc33.iaa.csic.es:18022/soda

PT meetings.

TODO: SODA response in datalink (DONE).

18/07/23 - Data ingest

Tested ingest data/metadata script, all OK. Left running in screen on src-dev.

Robustifying sited/datalink services with liveness probes.

Some talk with Coral/Orange about SODA.

TODO:

LONG TERM:

17/07/23 - Helping Coral / Data ingest

Ingest RACS data from SI (wrote script to download, now need to register and add metadata).

Helping Coral with dashboard visualiation of tests.

Download is running in screen on src-dev.

14/07/23 - A/L

13/07/23 - Orange support (timeouts)

Working a bit on data mgmt doc explaining group authz. Added returns.

Looked a bit at timeouts on upload. Looks like token expiry / requirement to specify a --transfer-timeout (default 360s).

Tried to make sited mongodb database persistent but getting:

  Warning  FailedAttachVolume  6s (x8 over 74s)  attachdetach-controller  AttachVolume.Attach failed for volume "pvc-ae2f35d8-2672-478d-b031-14ed8f1cfc56" : rpc error: code = Internal desc = [ControllerPublishVolume] Attach Volume failed with error failed to attach 58c7ea20-94b3-4c08-8e89-772b838ce51e volume to 693b8e3c-0c41-47b5-ab40-2e1dd6d8ee2c compute: Gateway Timeout

on 

ubuntu@srcnet-capi-management-1:/opt/src-services-deployment/site-directory$ helm-srcnet-workload-1 install -n site-directory --values values.yaml site-directory /opt/src-site-directory/etc/helm/

Remember to delete pv after deleting package via helm. Close to limit of volumes in openstack.

12/07/23 - Orange support (s_region)

Looking at issues with spoly column. Supporting Orange.

Comes down to fact that you need to select a data type for s_region. See https://wiki.ivoa.net/twiki/bin/view/IVOA/ObsLocTAP. If using spoly, you then need to approximate the circle with a polygon. You can't just expect it to parse an scircle f():

e.g. 

INSERT INTO rucio.dids (scope, name, vo, data) VALUES ('10', 'PTF10tce.fits', 'def', '{"s_region": "{(0,0),(1,0),(1,1)}", "obs_id": "id_4", "calib_level": 3, "obs_collection": "collection_4", "obs_publisher_did": "publisher_did_4"}');

works and upserts into obscore with s_region casted as an spoly, but 

SELECT * FROM jsonb_populate_record(null::rucio.obscore_row, '{"s_region": "scircle(''(1d,90d), 1d'')"}');

doesn't (note that this is the line in the upsert function that fails).

May be possible to have this scircle -> spoly casting in SQL, but that seems clumsy, e.g. in the upsert function:

--IF NEW.data ? 's_region' THEN
--  NEW.data = NEW.data - 's_region';
--  NEW.data = NEW.data || jsonb_build_object('s_region', scircle('<(1d,90d), 1d>'));
--END IF;

where you'd build the jsonb object by concatting the original with the new input field (note now not a string, it's an object).

11/07/23 - Meetings

10/07/23 - Datalake API

To get storage:*, needed to be added intop escape/data-manager group (part of scope api policy for ESCAPE IAM): https://indigo-iam.github.io/v/v1.7.2/docs/reference/api/scope-policy-api/

Also changed SA policy as follows:

https://github.com/italiangrid/storm-webdav/blob/master/doc/storage-area-configuration.md

# Enables read access to storage area files to users authenticated using OAuth/OIDC. Defaults to true.
orgsGrantReadPermission=false

# Enables write access to storage area files to users authenticated using OAuth/OIDC. Defaults to false.
orgsGrantWritePermission=false

# Enables scope-based authorization following the rules imposed by the WLCG JWT profile. Defaults to false.
wlcgScopeAuthzEnabled=true

# Enables fine-grained authorization engine. Defaults to false.
#fineGrainedAuthzEnabled=true

WLCG token policy config: https://wlcg-authz-wg.github.io/wlcg-authz-docs/token-based-authorization/

07/07/23 - A/L

06/07/23 - Datalake API

Tidying up, adding token cache, metadata validation.

05/07/23 - Datalake API

Working on token exchange, adding /locate

04/07/23 - Workshop

03/07/23 - Workshop prep


Top